Scaffolding: Federation sync app creation#298
Conversation
klpoland
added
feature
klpoland
changed the title
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
|
I'll add more comments after those are addressed to reduce the noise |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 3 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit d0608d2. Configure here.
| asset=asset, | ||
| asset_type=asset_type, | ||
| ) | ||
| await push_asset_updated_to_peers(http, config, payload) |
There was a problem hiding this comment.
Stale skip still notifies peers
High Severity
After a Redis event, handle_redis_asset_event always calls push_asset_updated_to_peers, even when FederatedAssetIndexer.apply_asset_event returns early because the event is stale. Peers can receive deletes or older updates that the local indexer intentionally ignored, causing cross-site index drift (for example after bootstrap sets a newer in-memory event_at).
Additional Locations (1)
Reviewed by Cursor Bugbot for commit d0608d2. Configure here.
| body = payload.model_dump(mode="json") | ||
| path = payload.asset_type.webhook_path | ||
| for peer in config.peers: | ||
| url = peer_webhook_url(peer, path) |
There was a problem hiding this comment.
Outbound sync ignores peer registry
Medium Severity
push_asset_updated_to_peers always uses static config.peers URLs from federation.toml, while site-hello updates PeerRegistry with each peer’s declared sync_service_url. Outbound webhooks never read the registry, so post-hello URL data is unused.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit d0608d2. Configure here.
| def _is_stale(self, site_name: str, uuid: UUID, event_at: datetime) -> bool: | ||
| key = doc_id(site_name, uuid) | ||
| prev = self._last_event.get(key) | ||
| return bool(prev is not None and event_at <= prev) |
There was a problem hiding this comment.
Naive-aware datetime compare crash
Medium Severity
Stale-event dedupe compares event_at from Redis (datetime.fromisoformat without normalizing timezone) against bootstrap/webhook timestamps that are timezone-aware UTC. If the gateway publishes naive ISO timestamps, Python raises TypeError on comparison and can crash indexing or the Redis subscriber loop.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit d0608d2. Configure here.
2 participants
In this PR:
federation/(v1) application with:services:bootstrap.py: for initialization of federated data from home site and peers (including when new peer is added)fed_index.py: for handling (external peer) asset indexing on webhook eventslocal_events.py: for handling redis dispatches to fetch local datapeer registry.py: for registering new peers and holding site information from site hellospeer_sync.py: for packaging and sending asset data to peer sites (through their webhook routes)routes:health.py: (stub for now) for pinging site health (is connection established?)webhooks.py: for receiving payloads FROM peers to index into LOCAL federated asset indices as well as asite-hellohook to confirm federation between peersschemasandmodels: for type checking and providing data structures to federation configs, events, and documents (how asset docs are represented byFederatedDatasetDoc,FederatedCaptureDocmay warrant further consideration re: anticipating schema flexibility)mainapp that subscribes to redis events for picking up signals from gateway (handled in PR Scaffolding: Federation gateway setup #299)Note
Medium Risk
New cross-site sync path writes to OpenSearch and accepts peer webhooks; mistakes in origin validation or indexing could propagate bad metadata, though scope is metadata-only and tests cover main flows.
Overview
Introduces a new
federation/FastAPI sync service that keeps federated dataset/capture metadata in OpenSearch and exchanges updates with peer sites.On startup it can bootstrap from local and peer gateway export APIs, register via
site-hello, and run a background Redis subscriber onfederation:events(gateway signals from a related change). Local events resolve public export docs, updatefed-*indices, and POST asset webhooks to configured peers; inbound/sync/api/v1/webhook/*routes apply the same indexing with allowlistedsite_namechecks.Repo wiring adds Docker/Compose, sample
federation.toml, dev mTLS cert tooling, and pre-commit hooks (federation-specific ruff/pyrefly/deptry). Coverage is a broad pytest suite (mesh integration, indexer regression, operational health).Reviewed by Cursor Bugbot for commit d0608d2. Bugbot is set up for automated code reviews on this repo. Configure here.